docs: add headless rendering troubleshooting guide to AGENTS.md#5389
docs: add headless rendering troubleshooting guide to AGENTS.md#5389pv-nvidia wants to merge 1 commit intoisaac-sim:developfrom
Conversation
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
There was a problem hiding this comment.
🤖 Isaac Lab Review Bot
Summary
This PR adds a "Headless Rendering" troubleshooting section to AGENTS.md, documenting common issues when running Isaac Lab with cameras/rendering in headless environments (DGX Cloud, containers, CI). The content is documentation-only with no code changes. The information appears accurate and addresses real pain points encountered during headless setup.
Architecture Impact
Self-contained. This is a documentation-only change to AGENTS.md (the project guidelines file). No code, APIs, or runtime behavior are affected.
Implementation Verdict
Ship it — Minor suggestions below for clarity, but nothing blocking.
Test Coverage
Not applicable — documentation-only PR. No tests needed.
CI Status
No CI checks available yet. For a documentation-only change, this is low risk.
Findings
🔵 Improvement: AGENTS.md:172 — Clarify the DISPLAY= syntax
The fix shows DISPLAY= ./isaaclab.sh -p -m pytest ... which sets DISPLAY to empty string for that command only. This is correct but might confuse readers unfamiliar with shell syntax. Consider adding a brief explanation:
# Set DISPLAY to empty for this command only
DISPLAY= ./isaaclab.sh -p -m pytest ...
# or unset it for the entire session
unset DISPLAY🔵 Improvement: AGENTS.md:206 — Link references itself
The note says "see PR discussions for current status" — this references the related PR mentioned in the description. However, once #5375 is merged, this link may become stale or confusing. Consider either:
- Removing the link if the
.kitfile fix will be documented elsewhere - Updating to reference the actual documentation/code location once the fix lands
🔵 Improvement: AGENTS.md:185-188 — Diagnosis commands could fail silently
The diagnosis commands (lsmod | grep, ls -la /dev/dri/renderD*, cat /sys/module/...) will return non-zero or produce confusing output if the modules/devices don't exist. Consider noting expected outputs for both working and broken cases:
# Check if nvidia_drm is loaded (empty output = not loaded)
lsmod | grep nvidia_drm
# Check for render nodes (should show renderD128, renderD129, etc.)
ls -la /dev/dri/renderD* 2>/dev/null || echo "No render nodes found"🔵 Improvement: AGENTS.md:163 — Consider adding a brief intro linking to related resources
The section jumps straight into problems/solutions. A one-liner intro noting that this supplements the official Isaac Sim headless documentation (if any exists) would help orient readers and avoid duplicating upstream docs.
🔵 Improvement: AGENTS.md:219-220 — Symlink pitfall could mention detection
The shared symlink issue is subtle and can cause confusing import errors. Consider adding how to detect this situation:
**Symptom**: `ImportError` or missing module errors after switching between repos that share `_isaac_sim`.
Overall this is solid, practical documentation that will save developers significant debugging time. The content accurately reflects real headless rendering challenges.
Greptile SummaryThis PR adds a "Headless Rendering" troubleshooting section to Confidence Score: 5/5Documentation-only change with accurate content; safe to merge. All findings are P2 style/completeness suggestions. No code is changed, all referenced scripts and kit files were verified to exist and match the documented behavior. No files require special attention. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Run isaaclab.sh] --> B{VIRTUAL_ENV set?}
B -- Yes --> C[Use venv Python]
B -- No --> D{CONDA_PREFIX set?}
D -- Yes --> E[Use conda Python\n⚠️ lacks Isaac Sim packages]
D -- No --> F{env_isaaclab/bin/python\nexists?}
F -- Yes --> G[Use env_isaaclab Python]
F -- No --> H{_isaac_sim/python.sh\nexists?}
H -- Yes --> I[Use Isaac Sim Python ✅]
H -- No --> J[Fallback: system python3]
E --> K[Fix: conda deactivate\nor CONDA_PREFIX= ./isaaclab.sh]
K --> I
Reviews (1): Last reviewed commit: "docs: add headless rendering troubleshoo..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| ### Conda conflicts | ||
|
|
||
| **Problem**: `isaaclab.sh` checks `CONDA_PREFIX` before `_isaac_sim/python.sh` — if conda is activated, it picks the conda Python which lacks Isaac Sim packages. |
There was a problem hiding this comment.
VIRTUAL_ENV precedence
The current isaaclab.sh checks VIRTUAL_ENV before CONDA_PREFIX (line 15–16 of the script). Describing it as "CONDA_PREFIX is checked before _isaac_sim/python.sh" is technically correct, but a developer who has both a venv and conda active will be confused when the venv wins instead of conda. Mentioning the full priority chain (VIRTUAL_ENV → CONDA_PREFIX → env_isaaclab → _isaac_sim) would make the section self-contained and accurate without needing readers to inspect the script.
| ### Missing `.kit` file dependencies for headless camera rendering | ||
|
|
||
| **Problem**: Camera annotators (RGB, depth, etc.) return empty data in headless mode because required extensions aren't loaded. | ||
|
|
||
| **Root cause**: The headless rendering experience file (`apps/isaaclab.python.headless.rendering.kit`) may be missing dependencies needed for the HydraTexture and viewport pipeline. | ||
|
|
||
| **Symptoms**: | ||
| - `rep.orchestrator.step()` hangs or loops infinitely | ||
| - Annotator `get_data()` returns None or zero-filled arrays | ||
| - Thousands of `createViewport` calls in Kit log | ||
|
|
||
| **Key extensions** needed for headless camera rendering: | ||
| - `omni.kit.hydra_texture` — Must be loaded before `omni.replicator.core` imports (availability is cached at import time) | ||
| - A viewport infrastructure extension — needed for render product creation | ||
|
|
||
| **Note**: `omni.kit.viewport.window` works but pulls in `omni.ui` which is heavyweight. Prefer a lighter alternative if available (see [PR discussions](https://github.com/isaac-sim/IsaacLab/pull/5375) for current status). |
There was a problem hiding this comment.
.kit section stops short of a concrete fix
The section identifies root cause and symptoms well, but the "Key extensions" list tells readers what is needed without explaining how to add it. Since apps/isaaclab.python.headless.rendering.kit already exists in the repo and is missing omni.kit.hydra_texture in its [dependencies] block, readers will likely end up editing the kit file incorrectly (e.g., adding it after omni.replicator.core when it must come before). A one-line example of the correct dependency ordering in the kit file would prevent that mistake.
Document common headless rendering issues for DGX Cloud, containers, and CI environments: - DISPLAY variable causing GLXBadFBConfig crashes - EGL vs Vulkan diagnosis steps - Missing .kit file dependencies for camera annotators - Conda conflict with Isaac Sim Python - Shared _isaac_sim symlink pitfall
pv-nvidia
force-pushed
the
docs/headless-rendering-guide
branch
from
42d1321 to
d78afd4
Compare
AntoineRichard
left a comment
AntoineRichard
left a comment
There was a problem hiding this comment.
Make the Agents.md changes leaner.
There was a problem hiding this comment.
I feel like some of these could be moved to troubleshooting of the isaaclab doc. And the agent.md could reference that part of the doc. This would avoid having an Agent.md that's too large. Other teams try to keep this file lean. If you feel otherwise let me know.
2 participants
Summary
Adds a "Headless Rendering" section to AGENTS.md documenting common issues and fixes when running Isaac Lab with cameras/rendering in headless environments (DGX Cloud, containers, CI).
Topics covered:
DISPLAYvariable:DISPLAY=:99causes GLXBadFBConfig crashes; fix: unset iteglInitializefails; Vulkan works without EGL.kitdependencies: Why camera annotators return empty data headlessly; key extensions neededisaaclab.shpicks wrong Python when conda is active_isaac_simsymlink: Multiple repos overwriting each other's pip packagesContext
These issues were encountered while setting up headless camera rendering on DGX Cloud for Fabric backend testing (related: #5375).
Note: The specific
.kitfile fix for missing extensions is being handled separately by @pbarejko with a lighter dependency thanomni.kit.viewport.window.